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(54) Method and apparatus for classification-adaptive data processing 



(57) A data processing apparatus includes a deter- 
mining unit capable of determining, in accordance with 
a plurality of data to be extracted from said input data, 
irregular intervals among said plurality of data to be 
extracted from said input data, an extracting unit for 
extracting the plurality of data, which correspond to out- 



put data of interest to be predicted, from said input data 
in accordance with the determination result by said 
determining unit, and a predicting unit for finding pre- 
dicted values of the output data of interest based on the 
plurality of data extracted by said extracting unit. 
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[0001] The present invention relates to data processing apparatuses and methods, learning apparatuses and 
methods, and media, and in particular, to a data processing apparatus and method for increasing processing efficiency 
when image data, etc., are processed, and to a medium provided with the same. 

[0002] The assignee of the present invention proposed classification adaptive processing as processing for 
increasing image quality, etc., and for improving other image features. 

[0003] The classification adaptive processing consists of classifying processing and adaptive processing. In the 
classifying processing, data are classified based on their properties, and adaptive processing on the data in each class 
is performed. The adaptive processing Is the following technique. 

[0004] In the adaptive processing, by linearly linking, for example, pixels (hereinafter refen-ed to as "input pixels") 
constituting an input image (an image to be processed by the classification adaptive processing) and predetermined 
prediction coefficients, predicted vailues of pixels of an original image (e.g., an image including no noise, an image free 
from blurring, etc.) are found, whereby an image in which noise included in the input image is eliminated, an image in 
which blurring generated In the input image is reduced, etc.. can be obtained. 

[0005] Accordingly, it is, for example, assumed that the original image is teacher data, and it is also assumed that 
an image obtained by superimposing noise on or blurring the original image is student data. The finding of predicted 
values E[y] of pixel levels y of pixels constituting the original image by using a linear first-order linking model defined by 
linear linking of a set of a plurality of student data (pixel levels) x^, X2, etc., and predetermined prediction coefficients 
w^, W2, etc., is considered. In this case, the predicted values E[y] can be represented by the following expression: 

E[y] = w^x^ + WgXg + • • • (1) 

[0006] For generalizing expression (1), by using the following expressions: 
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to define a matrix W composed of a set of prediction coefficients w, a matrix X composed of a set of student data, and 
a matrix Y' composed of a set of predicted values E[y], the following observation equation holds:. 

XW = Y' (2) 

[0007] Here, a component Xy of the matrix X represents the j-th student data in an i-th set of student data (a set of 
student data for use in prediction of the i-th teacher data y^, and a component wj of the matrix W represents a prediction 
coefficient in which the product of the coefficient and the j-th student data in the set of student data is calculated. A com- 
ponent yj represents the j-th teacher data, and E[y|] accordingly represents a predicted value of the j-th teacher data. 
[0008] The finding of each predicted value E[yl close to each pixel level y of the original pixel by applying a least 
square operation to the observation equation (2) is considered. In this case, by using the following expressions: 
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to define a matrix Y composed of a set of actual pixel levels y of the original pixels which are used as teacher data, and 
a matrix E composed of a set of residuals e of predicted values E(y] from the pixel levels y of the original pixels, the fol- 
lowing residual equation holds from expression (2): 



XW = Y + E 



(3) 



[0009] In this case, the prediction coefficients w for the predicted values E[y] close to the pixel levels of the original 
pixels can be found by minimizing the following squared error: 



20 



. 25 [0010] Therefore, when the result of differentiating the squared errors with respect to the prediction coefficient Wj is 
zero, the prediction coefficient Wj, which satisfies the following expression, is an optimal value for finding the predicted 
values E[y] close to the pixel levels y of the original pixels. 
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[0011] Accordingly, by using the prediction coefficient W| to differentiate expression (3), the followinjg expressions 
hold: 
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4o [0012] From expressions (4) and (5), the following expressions are obtained: 

. m . m rn- 

£ejXii=0,£e|Xi2=0, .E®i^in = 0 

i=1 i=1 r i=1 
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[001 3] When relationships among the student data x, the prediction coefficients w, the teacher data y, and the resid- 
uals e, are taken into consideration, the following normalization equations can be obtained from expression (6). 
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[0014] By preparing a certain number of student data x and a certain number of teacher data y, the normalization 
equations (7) can be fomied corresponding to the number of prediction coefficients w. Accordingly, by solving equations 
(7) (although the solution of equations (7) requires a matrix composed of coefficients on the prediction coefficients w to 
be regular), the optimal prediction coefficients w can be found. For solving equations (7), Gauss-Jordan elimination, or 
the lil^e, may be used. 

[0015] The adaptive processing is the above-described processing in which the predicted values E[y] close to the 
pixel levels y of the original pixels are found based on expression (1), using the prediction coefficients w found before- 
hand. 

[0016] The adaptive processing differs from, for example, simple interpolation in that components which are not 
included in an input image but which are included in the original image are reproduced. In other words, as long as atten- 
tion is paid to only expression (1), the adaptive processing is the same as intisrpolation using a so-called interpolating 
filter. However, in the adaptive processing, the prediction coefficients w corresponding to tap coefficients of the interpo- 
lating filter are obtained by a type of leaming using the teacher data y for each class! Thus, components included in the 
original image can be reproduced. That is, an image having a high signal-to-noise ratio can be easily obtained. From 
this feature, it may be said that the adaptive processing has an image creating (resolution creating) operation. There- 
fore, in addition to the case where predicted values of an image obtained by eliminating noise and blurring in the original 
image are found, the adaptive processing may be used for, for example, the conversion of an image having a low or 
standard resolution into an image having a high resolution. 

[0017] As described above, in the classification adaptive processing, the adaptive processing is perfomried for each 
class. In classification performed before the adaptive processing, a plurality of input pixels are extracted which are adja- 
cent to each of original pixels (hereinafter referred to also as "onginal pixels of interest") based on which a predicted 
value is found, and the original pixels of interest are classified based on the characteristics of the input pixels (fe.g., pat- 
terns of the input pixels, an inclination of the pixel levels, etc.). Input pixels fixedly positioned with respect to the original 
pixels of interest are extracted as the plurality of input pixels for the classification. 

[0018] Nevertheless, In cases where the classification adaptive processing is used to convert, for example, a 
blurred input image into a blur-reduced image, the use of input pixels fixedly positioned to original pixels of interest to 
classify the original pixels of interest regardless of the degree of the blurring in the input image may make it difficult to 
perfomri classification that sufficiently reflects the characteristics of the original pixels of interest. 
[0019] By way of example, when classification adaptive processing on an input image having a small degree of blur- 
ring is performed, by preferably performing, in an aspect of image correlation, classification using input pixels relatively 
close to the original pixels of interest, classification that reflects the characteristics of the original pixels of interest can 
be perfbmied. When classification adaptive processing on an input image having a large degree of blun-ing is per- 
formed, by preferably performing. In an aspect of adverse effects of the blurring, classification using input pixels rela- 
tively far from the original pixels of interest, classification that reflects the characteristics of the original pixels of interest 
can be performed. 

[0020] Accordingly, the use of the input pixels fixedly positioned to the original pixels of interest for classification of 
the original pixels of interest may hinder classification of the original pixels of interest, which reflects their characteris- 
tics. As a result, the performance of the classification adaptive processing may deteriorate, that is, an innage (here an 
image obtained by sufficiently reducing blur) obtained by using the classification to sufficiently improve the input image 
may not be obtained. 

[0021] In the adaptive processing, input pixels fixedly positioned to pixels of interest ar^ used to perfomri computa- 
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tion based on the linear prediction expressions (1 ), whereby predicted values based on pixels of interest are found. Also 
in this case, it is expected, similarly to the case of the classification that, by preferably performing computation based 
on the linear prediction expressions (1) with input pixels variably positioned to the pixels of interest, as required, pre- 
dicted values having small prediction errors from the pixels of interest can be found. 
5 [0022] Various respective aspects of the invention are defined in the appended claims. 

[0023] Embodiments of the present invention to provide a data processing apparatus and method that solves the 
foregoing problems, thereby increasing processing efficiency when image data, etc., are processed, and a medium pro- 
vided with the same 

[0024] To this end, according to an aspect of the present invention, there is provided a data processing apparatus 
10 for predicting output data corresponding to input data by processing the input data. The data processing apparatus 
includes a determining unit capable of determining, in accordance with a plurality of data to be extracted from the input 
data, irregular intervals among the plurality of data to be extracted from the input data, an extracting unit for extracting 
the plurality of data corresponding to output data of interest to be predicted from the input data in accordance with the 
determination result by the determining unit, and a predicting unit for finding predicted values of the output data of inter- 
15 est based on the plurality of data extracted by the extracting unit. 

[0025] According to another aspect of the present invention, there is provided a learning apparatus for predicting 
output data corresponding to input data by processing the input data. The learning apparatus includes a determining 
unit capable of determining, in accordance with a plurality of data to be extracted from the input data, irregular intervals 
among the plurality of data to be extracted from the input data, and an extracting unit for extracting the plurality of data 
20. corresponding to output data of interest to be predicted from the input data in accordance with the determination result, 
and a computing unit for finding prediction coefficients based on the extracted data. 

[0026] According to a further aspect of the present invention, there is provided a data processing method for pre- 
dicting output data corresponding to input data by processing the input data. The data processing method includes the 
steps of enabling determination of irregular intervals among a plurality of data to be extracted from the^ input data in 
.25 accordance with the plurality of data to be extracted, extracting the plurality of data corresponding to output data of 
interest to be predicted from the input data in accordance with the determination result, and finding predicted values of 
the output data of interest based on the extracted plurality of data. 

[0027] According to a still further aspect of the present invention, there is provided a learning method for predicting 
output data corresponding to input data by processing the input data. The learning method includes the steps of ena- 
30 bling determination of irregular intervals among a plurality of data to be extracted from the input data in accordance with 
the plurality of data to be extracted, extracting the plurality of data corresponding to output data of interest to be pre- 
dicted from the input data in accordance with the deterrnination result, and finding prediction coefficients based on the 
extracted plurality of data. 

[0028] According to yet another aspect of the present invention, there is provided a storage medium containing a 
35 computer-controllable program for predicting output data corresponding to input data by processing the input data. The 

computer-controllable program includes the steps of enabling determination of irregular intervals among a plurality of 
. data to be extracted from the input data in accordance with the plurality of data to be extracted, extracting the plurality 

of data corresponding to output data of interest to be predict efel from the input data in accordance with the determination 

result, and finding predicted values of the output data of interest based on the extracted plurality of data. 
40 [0029] According to yet another aspect of the present invention, there is provided a storage medium containing a 

computer-controllable program for predicting output data corresponding to input data by processing the input data. The 

computer-controllable program includes the steps of enabling determination of irregular Intervals among a plurality of 
. data to be extracted from the input data in accordance with the plurality of data to be extracted, extracting the plurality 

of data corresponding to output data of interest to be predicted frpm the input data in accordance with the determination 
45 result, and finding prediction coefficients based on the extracted plurality of data. 

[0030] The invention will novy be described by way of example with reference to the accorhpanying drawings, 

throughout which like parts are referred to by like references, and in which: 

Fig. 1 is a block diagram showing an image processing apparatus according to an. embodiment of the present 
50 invention; 

Fig. 2 Is an illustration of processing by the image processing apparatus shown in Fig. 1 ; 

Fig. 3A Is an Illustration of a tap composed of 3 x 3 pixels, output from the tap-determining circuit 7 shown in Fig. 
1 , which has a tap width of zero; 

Fig. 3B is an illustration of a tap composed of 3 x 3 pixels, output from the tap-determining circuit 7 shown in Rg: 
55 1, which has a tap width of one; 

Rg. 4 is a flowchart illustrating a blur-reducing process by the image processing apparatus shown in Rg. 1 ; 
Fig. 5 is a block diagram showing the tap-determining circuit 7 shown in Fig. 1 ; 

Fig. 6 Is a flowchart illustrating a tap-determining process by the tap-determining circuit 7 shown In Fig. 5; 
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Fig. 7 is a block diagram showing a first embodiment of the image processing apparatus to which the present inven- 
tion IS applied; 

Fig. 8 is a flowchart illustrating a learning process by the learning apparatus shown in Fig. 7; 
Fig, 9 is a block diagram showing the tap-determining circuit 72 shown in Fig. 7; 
5 Fig. 10 is a flowchart Illustrating a tap-determining process by the tap-determining circuit 72 shown in Fig. 9; 

Fig. 11 is a graph showing relationships between a tap width and the average of standard deviations of taps, which 
are used for processing by the reference-vatue generator 84 shown in Fig. 9; 

Fig. 12 is a graph showing relationships between a tap width and signat-to-ndise ratios of predated values of taps, 
which are used for processing by the reference-value generator 84 shown in Fig. 9; 
10 Fig. 1 3 is an illustration of a basic tap set by the tap-determining circuit 7 In a second embodiment of the present 
invention; 

Fig. 14 Is an illustration of directions from the center pixel as a start point which are detected by the tap-detemnin- 
ing circuit 7 shown in Fig. 1 ; 

Fig. 15 is an illustration of a tap arrangement in which taps are moved in accordance with waveform characteristics 
15 of the directions shown in Fig. 1 4 by the tap-determining circuit 7 shown in Fig. 1 ; 

Fig. 1 6 is a flowchart illustrating a blur-reducing process by the image processing apparatus shown in Fig. 1 ; 
Fig. 1 7 is a flowchart illustrating a first modification of a tap-determining process In step S41 shown in Fig. 1 6; 
Fig. 1 8A is a graph showing a tap arrangement initially obtained by the first modification of tap-determining process 
in step S41 shown in Fig. 16; 

20 Fig. 18B is a graph showing a tap arrangement as a result of the first modification of the tap-determining process 
in step S41 shown in Fig, 16; 

Fig. 1 9 is a flowchart illusttBting a second modification of the tap-determining process in step S41 shown In Fig. 1 6; 
Fig. 20A is a graph showing a tap arrangement initially obtained by the second modification of the tap-determining 
process in step S41 shown in Fig. 16; 
25 Fig. 20B is a graph showing a tap arrangement as a result of the second modification of the tap-determining proc- 
ess In step S41 shown In Fig. 16; 

Fig. 21 is a flowchart illustrating a third modification of the tap-determining process in step S41 shown in Fig. 1 6; 
Fig. 22A is a graph showing a tap arrangement initially obtained by the third modification of the tap-determining 
process in step S41 shown in Fig. 16; 
30 Fig. 22B is a graph showing a tap arrangement as a result of the third modification of the tap-determining process 
in step S41 shown in Fig. 16; 

Fig. 23 is a block diagram showing a second embodiment of the learning apparatus of the present invention; 
Fig. 24 is a flowchart illustrating a learning process by the learning apparatus shown in Fig. 12; 
Fig. 25A is an illustration of a computer 101 into which a program for executing processing according to the present 
35 invention is installed; 

Fig. 258 is an illustration of types of a recording medium for storing a program for executing processing according 
to the present invention; 

Fig. 25C is a concept illustration of distribution of a prograrti for executing processing according to the present 
invention to a computer via a satellite or a network; and 
40 Fig. 26 is a block diagram showing an example of a computer into which a program for executing processing 
according to the present invention is installed. 

[0031] Rg. 1 shows a first embodiment of an image processing apparatus to which the present invention is applied. 
[0032] When, for example, blurred images are input as input images to the image processing apparatus, the image 
45 processing apparatus can perform classification adaptive processing on the input images, thereby outputting images 
(blur-reduced image) in which the blurring is sufficiently reduced regardless of the degree of blurring in the input 
images, as shown in Fig. 2. 

[0033] The image processing apparatus includes a frame memory 1 , class-tap generating circuit 2, a prediction-tap 
generating circuit 3, a classifying circuit 4, a coefficient random access memory (RAM) 6, a prediction arithmetic circuit 
so 6, and a tap-detennining circuit 7. Input images to be processed for reducing blurring are input to the image processing 
apparatus. 

[0034] The frame memory 1. temporarily stores the image input to the image processing apparatus in, for example, 
units of frames. In the first embodiment, the frame memory 1 can store input images as a plurality of frames by bank 
switching. The frame menhory 1 can perform a process of temporarily storing images input to the image processing 
55 apparatus in units of frames, even if the input images are moving pictures. 

[0035] Using original pixels (ideal blurless pixels obtained by completely eliminating blurring in input pixels in the 
first embodiment) based on which predicted values are found by classification adaptive processing, as original pixels of 
interest the class-tap generating circuit 2 extracts input pixels for classification of the original pixels of interest from the 
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input images stored in the frame memory 1 in accordance with tap information supplied from the tap-determining circuit 
7, and outputs the extracted pixels as class taps to the classifying circuit 4. 

[0036] The prediction tap generating circuit 3 extracts the input pixels for finding, by the prediction arithmetic circuit 
6, predicted values based on the original pixels of interest in accordance with the tap information from the tap-determin- 

5 ing circuit 7, and supplies the extracted pixels as prediction taps to the prediction arithmetic circuit 6. 

[0037] The classifying circuit 4 classifies the original pixels of interest based on the class taps from the class-tap 
generating circuit 2, and supplies class codes corresponding to classes obtained by the classification, as addresses, to 
the coefficient RAM 6. In other words, the classifying circuit 4 performs, for example, 1 -bit adaptive dynamic range cod- 
ing (ADRC) on the class taps from the class-tap generating circuit 2, and outputs the obtained ADRC codes as class 

10 codes to the coefficient RAM 5. 

[0038] Here, in k-bit ADRC, from among the input pixels constituting the class taps, the maximum value MAX and 
the minimum value MIN are detected, a value DR = MAX - MIN is used as a dynamic range, and based on the dynamic 
range DR, the input pixels constituting the class taps are requantized to have k bits. In other words, from the pixel levels 
of the pixels constituting the class taps, the minimum value MIN is subtracted, and the obtained values are divided by 

15 (quantized). Accordingly, when the class taps are processed by l-bit ADRC, the pixel level of each of the input 

pixels has one bit. In this case, a string of bits obtained by arranging 1-bit pixel levels based on the pixels constituting 
the class taps in predetermined order, as described above, is output as an ADRC code. 

[0039] The coefficient RAM 5 stores prediction coefficients for classes, which are obtained such that a learning 
apparatus (described below) performs learning. When being supplied with a . class code from the classifying circuit 4, 
20 the coefficient RAM 5 reads prediction coefficients stored at an address corresponding to the class code, and supplies 
the code to the prediction arithmetic circuit 6. 

[0040] The prediction arithmetic circuit 6 finds the predicted values E[y] based on the original pixels y of interest by 
performing computation based on expression (1) with the prediction coefficients w1, w2, etc., supplied from the coeffi- 
cient RAM 5, and the pixel levels x1 , x2, etc., supplied from the prediction -tap generating circuit 3, which constitutes the 
25 prediction taps. The prediction arithmetic circuit 5 outputs the predicted values E[y] as the pixel levels of pixels in which 
blurring is eliminated. 

[0041] The tap-determining circuit 7 determines, based on a statistic stored in the frame memory 1, a plurality of 
input pixels constituting class taps and prediction taps, and supplies information (hereinafter referred to also as "tap 
infomnation") about the class taps and the prediction taps to the class-tap generating circuit 2 and the prediction-tap 
30 generating circuit 3. 

[0042] In other words, the tap-determining circuit 7 outputs the tap information, which basically controls the class - 
tap generating circuit 2 and the prediction-tap generating circuit 3 to generate square class taps and prediction taps 
(hereinafter both referred to simply as "taps") each composed of, for example, three horizontal pixels and three vertical 
pixels in which an input pixel positioned corresponding to an original pixel of interest is used as a center pixel. The tap 
35 infonnation controls the interval between pixels constituting the tap to differ depending on the statistical value of the 
input images. 

[0043] Specifically, when the statistical value of the input images is, for example, a certain value, the tap-determin- 
ing circuit 7 outputs tap information for generating a tap whifch is composed of 3 x 3 pixels around a center pixel and 
which has a tap width of zero (the interval between pixels constituting a tap is zero), as shown in Rg. 3A. When the sta- 
40 tistical value of the input images is, for exaniple, another value, the tap-deterrhining circuh 7 outputs tap information for 
generating a tap which is composed of 3 x 3 pixels around a center pixel and which has a tap width of one (the interval 
between pixels constituting a tap is one pixel or one frame), as shown in Fig. 3B. 

[0044] Referring to the flowchart shown in Fig. 4, a process for reducing blunring in the input images, which is per- 
formed in the image processing apparatus shown in Fig. 1 , is described below. 

45 [0045] Input images (moving picture) to be processed so that blurring-reduction is performed are supplied to the 
frame memory 1 in units of frames. The input images are sequentially stored in units of frames in the frame memory 1 . 
[0046] In step SI , the tap-determining circuit 7 determines, based on the statistical value of the input images stored 
in the frame memory 1, a plurality of input images for generating taps, and outputs tap inforrnation about the input 
images to the class-tap generating circuit 2 and the prediction-tap generating circuit 3. 

so [0047] When receiving the tap information, in step S2, in accordance with the tap information, the class-tap gener- 
ating circuit 2 and the prediction-tap generating circuit 3 each read, from the frame rriemory 1 , a plurality of input images 
for generating a class tap and a prediction tap for each original pixel of interest based on which a predicted value is 
found, and generates the class tap and the prediction tap. The class tap and the prediction tap are supplied to the clas- 
. sifying circuit 4 and the predrction arithmetic circuit 6, respectively. : 

55 [0048] When receiving the class tap from the class-tap generating circuit 2, in step S3, the classifying circuit 4 clas- 
sifies the original pixels of interest based on the received class tap. The classifying circuit 4 outputs obtained class 
codes as addresses to the coefficient RAM 6. In step S4, the coefficient RAM 5 reads a prediction coefficient stored at 
ah address corresponding to the class code from the classifying circuit 4, and supplies the prediction coefficient to the 



BNSDOQD: <E P 1061473A1 I > 



EP 1 061 473 A1 



prediction arithmetic circuit 6. 

[0049] In step S5, the prediction arithmetic circuit 6 uses both the prediction tap from the prediction-tap generating 
circuit 3 and the prediction coefficient from the coefficient RAM 5 to perform computation based on expression (1), 
thereby finding the predicted value E[yl based on the original pixel y of interest. In other words, a blurring-reduced pixel 
is found, and the process proceeds to step S6. In step S6, the prediction arithmetic circuit 6 outputs the predicted value 
E[y] based on the original pixel y of interest obtained in step S5 as a pixel level in which bluning is reduced at an input 
pixel having the same position as the original pixel y of interest. The process proceeds to step S7. 
[0050] In step S7, a controller or the like detemnines whether the above-described processing on all original pixels 
of interest in a predetermined block (described below) has been perfomied. If the controller has determined negatively, 
the process proceeds back to step S2. In step S2. among the original pixels in the block, those unused as original pixels 
of interest are used as further original pixels of interest, and the same processing is subsequently performed. Accord- 
ingly, in the first embodiment, for original pixels of interest in the same block, a tap is generated based on the same tap 
information. In other words, for the original pixels of interest in the same block, a tap is generated from input pixels iden- 
tically positioned with respect to each original pixel of interest 

[0051] If the controller has detemnined in step S7 that the process has been performed using all the original pixels 
in the predetermined block as original pixels of interest, the process proceeds to step S8, In step SB, the controller or 
the like determines whether there is a block to be subsequently processed, in other words, whether an input image cor- 
responding to the block to be subsequently processed is stored the frame memory 1. If the controller has determined 
in step SB that the input image corresponding to the block to be subsequently processed is stored in the frame memory 
1 . the process returns to step SI , and similar processing is subsequently perfonmed. Accordingly, new tap infomnation 
is determined for the block to be subsequently processed, and a tap is generated In accordance with the tap infomna- 
tion. 

[0052] If the controller or the like has determined in step SB that the input image con^sponding to the block to be 
subsequently processed is stored in the frame memory 1 , the process is terminated. 

[0053] Referring to Fig. 5, an example of the tap-detemiining circuit 7 shown in Fig. 1 is described below. 
[0054] A reading unit 11 reads, when an original pixel in the predetermined block is used as an original pixel of inter- 
est, input pixels to be used as a tap based on the original pixel of interest from the frame memory 1 (shown in Fig. 1), 
and supplies the pixels to a standard-deviation calculator 12. 

[0055] In the reading unit 1 1 , one frame of an original inriage, regions obtained by dividing the one frame, or several 
frames (e.g., a frame just after change of scenes to a frame just before the next change of scenes) are used as one 
block, and tap information is determined in the block unit. When each original pixel in a block is used as an original pixel 
of interest, the reading unit 1 1 reads, for each original pixel, input pixels constituting a tap having a tap width in accord- 
ance with control by a determining unit 13 from the frame memory 1, and supplies the pixels to the standard-deviation 
calculator 12. 

[0056] The standard-deviation calculator 12 calculates, as a statistical value of the tap from the reading unit 1 1 . a 
standard deviation (accurately a value equivalent to the standard deviation) of the pixel levels of the input pixels consti- 
tuting the tap. The standard-deviation calculator 12 calculates an average score of standard deviations of taps having 
the same tap width, and supplies the determining unit 1 3 with the Average score as a value of evaluating each tap width 
for each block. When a block is represented by i, and a tap width is represented by j. the standard-deviation calculator 
12 calculates score(i,j) represented by the following expression as a value of evaluating a tap width j, and the score(i,j) 
to the determining unit .13. 



k=1 



where M represents the number of original pixels in a block #i. K represerits the number of input pixels constituting a 
tap, j K represents the pixel level of the k-th inptit pixel in a tap hiaving a tap width j, which corresponds to the m-th 
original pixel in a block #i, and mean^j represents an average value of taps having the tap width j, which corresponds 
to the m-th pixel in the block #1, Therefore, mean^j can be found by performing an operation of. 

(Vmj.1+V„j^2 + ... + V^j,,)/k. . ^ 

[0057] The determining unit 13 conipares the evaluated value score(i,J) from the standard-deviation cafoulator 12 
and a predetermined reference va|ue set in a reference-value setting unit 14, and controls the reading unit 1 1 based on 
the comparison result. Based on the comparison result, the determining unit 13 detemnines a tap width used when a 
tap is generated for original pixels in the block (hereinafter refei-red to also as the "block of interest") including the orig- 
inal pixels of interest, and outputs the tap width as tap information. 

[0058] In the reference-value setting unit 14, a standard deviation of pixel levels of input pixels (student data) con- 



. <EPj 1061473A1J_> 



EP 1 061 473 A1 

stituting a tap in the case where blurring in the input images is most reduced (in the case where the result of classifying 
input pixels is closest to original pixels corresponding to the input pixels) is set as a reference value for comparing two 
outputs from the standard-deviation calculator 12. The way of finding the reference value is described later. 
[0059] Referring to the flowchart shown in Fig. 6, a tap determining process is below described in which the tap- 

5 determining circuit 7 shown in Fig. 5 determines tap structure: 

[0060] In the tap detemnining process, when any one of the original pixels in the block of interest is initially used as 
an original pixel of interest, in step S11. from the frame memory 1 (shown in Fig. 1), the reading unit 11 reads, for the 
original pixel used as an original pixel of interest in the block of interest, for example, input pixels for generating taps 
having a tap width of zero, and supplies the tiaps to the standard-deviation calculator 12. 

10 [0061] The process proceeds to step S12. In step S12, the standard-deviation calculator 12 calculates a standard 
deviation of the taps composed of the input pixels, and the process proceeds to step S13. In step SI 3, the controller or 
the Hke determines whether the above-described processing on all pixels in the block of interest has been performed. 
If the controller or the like has determined negatively, the process returns to step S1 1 . From among the original pixels 
in the block of interest, those that have not been used are used as further original pixels of interest, and similar process- 

75 ing is repeatedly perfomied. 

[0062] If the controller or the like has determined in step SI 3 that the above-described prociessing on alt the original 
pixels in the block of interest has been perfomied, the process proceeds to step S14. In step 814. the standard-devia- 
tion calculator 12 calculates the average of the obtained tap standard deviations for the original pixels in the block of 
interest 

20 [0063] In other words, when the block of interest is represented by block #i, and the tap width of a tap generated for 
each original pixel in the block #i of interest is represented by j. the score(i.j) represented by expression (8) is calculated 
in step SI 4. The score(i,j) is supplied to the detemnining unit 13. 

[0064] In step S15, the detennining unit 1 3 determines whether the score(i,j) from the standard-deviation calculator 
12 is close to the reference value set in the reference-value setting unit 14. If the determining unit 13 has detemriined 
25 negatively, i.e., when the average of standard deviations of taps having a tap width j. which are generated for original 
pixels in the block of interest #1 , is not close to the reference value, the process proceeds to step SI 6. in step SI 6, the 
determining unit 13 controls the reading unit 11 to change the tap width j, and the process proceeds back to step S1 1 . 
[0065] In this case, in step 81 1, the reading unit 1 1 uses a certain original pixel in a block of interest as an original 
pixel of interest, and reads, for the original pixel of interest, input pixels for generating taps having a tap width j+1 from 
30 the frame memory 1 . Subsequently, similar processing is performed. 

[0066] When, in step SI 5, the determining unit 1 3 determines that the score(i,j) from the standard-deviation calcu- 
lator 12 is close to the reference value set in the reference-value setting unit 14, that is, when the average of standard 
deviations of taps (having the tap width j) generated for the original pixels of the block #1 of interest is close to the ref- 
erence value, the process proceeds to step SI 7. In step S1 7, the determining unit 1 3 outputs the tap width j as tap infer- 
os mation, and the process returns. 

[0067] As described above, in accordance with tap information for each block, which is output by the tap-detemriin- 
ing circuit 7, the class-tap generating circuit 2 and the prediction -tap generating circuit 3 generate taps in which original 
pixels in each block are used as original pixels of interest. When the tap-determining circuit 7 outputs tap information 
based on the tap width j. taps having the tap width j output as the tap information are generated for the original pixels 
40 constituting the block of interest. ^ 

[0068] Refen-ing to Fig. 7, a first type of learning apparatus according to an embodiment of the present invention is 
below described which finds prediction coefficients for classes to be stored in the coefficient RAM 5 (shown in Fig. 1) 
and which finds a predetemnined reference value to be set in the refers nee -value setting unit 14. (shown in Fig. 5). 
[0069] A frame memory 61 is supplied with original images .(here, blurless images) to be used as the teacher data 
45 y in, for example, units of frames. The original images are tehiporarily stored in the franrie nriemory 61 . A blumng circuit 
62 generates blurred images (as the student data x) by reading the original images (to be used as the teacher data y in 
the learning of prediction coefficients) stored in the frame memory 61 . and blurring original pixels constituting the orig- 
inal images (e.g., using a low-pass filter to perform filtering). The blurred images are supplied to a frame memory 63. 
[0070] The frame memory 63 temporarily stores the blurred images from the bluning circuit 62. 
50 [0071] The frame memories 61 and 63 have a structure identfcal to that of the frame memory 1 . 

[0072] A class-tap generating circuit 64 and a prediction-tap generating circuit 65 use the blurred-image constitut- 
ing pixels (hereinafter referred to as the "blurring pixels") stored in the frame memory 63 to respectively generate a class 
tap and a prediction tap for each original pixel of interest in accordance with tap information from a tap-determining cir- 
cuit 72, and respectively supplies the class tap and the prediction tap to a classifying circuit 66 and an adder circuit 67, 
55 similariy to the class-tap generating circuit 2 and the predtetion-tap generating circuit 3 shown in Rg. 1 . 

[0073] / The classifying circuit 66 has a structure identical to that of the classifying circuit 4 shown in Rg. 4. The clas- 
sifying circuit 66 classifies, based on the class tap from the class-tap generating circuit 64, the original pixels of interest, 
and supplies the corresponding dass code as an address to a prediction-tap memory 68 and a teacher-data memory 
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70. 

[0074] The adder circuit 67 performs computation corresponding to summation {Z) as a multiplier for the prediction 
coefficient w in each left side of the normalization equations (7) by reading the stored address con-esponding to the 
class code output by the classifying circuit 66 from the prediction-tap memory 68, and adding the read value to the pixel 
5 level of each blurring pixel constituting the prediction tap. which is supplied from the prediction«tap generating circuit 65. 
The adder circuit 67 overwrites the computation result at an address in the prediction-tap memory 68, which corre- 
sponds to the class code output by the classifying circuit 66. 

[0075] The prediction-tap memory 68 reads the stored address corresponding to the classifying circuit 66, supplies 
the read address to the adder circuit 67, and stores an output value from the adder circuit 67 at the supplied address. 

10 [0076] The adder circuit 69 reads an original pixel of interest as a teacher data y from among the original pixels con- 
stituting the original images, which are stored in the frame memory 61 , and reads an address stored in the teacher-data 
memory 70, which corresponds to the class code output from the classifying circuit 77. The adder circuit 69-adds the 
value read from the teacher-data memory 70 and the teacher data (original pixel) y read from the frame memory 61, 
thereby perfomriing computation corresponding to summation (L) in each right side of the normalization equations (7). 

15 The adder circuit 69 overwrites the computation result at an address in the teacher-data memory 70, which con-e- 
sponds to the class code output from the classifying circuit 70. 

[0077] The adder circuits 67 and 69 also perfonn multiplication in expressions (7). Specifically, the adder circuit 67 
perfonns multiplication of blurring pixels x constituting the prediction tap, and the adder circuit 69 performs multiplication 
of a blurring pixel x and the teacher data y Accordingly, the blumng pixel x, required by the adder circuit 69, is read from 
20 the frame memory 63. 

[0078] The teacher-data memory 70 reads the address stored corresponding to the class code output from the 
classifying circuit 66, supplies the read address to the adder circuit 69, and stores an output from the adder circuit 69 
at the supplied address. 

[0079] The computing circuit 71 can sequentially read the values stored at addresses corresponding to the class 

25 codes in the prediction-tap memory 68 and the teacher-data memory 70, and can solve the nomrialization equations 
(7), which uses the read values, whereby prediction coefficients for the classes can be found. In other words, the com- 
puting circuital fonns the nomnaiization equations (7) based on the values stored at the addresses corresponding to 
the classes in the prediction-tap memory 68 and the teacher-data memory 70, and solves the equations, thereby finding 
the prediction coefficients for the classes. 

30 [0080] The tap-determining circuit 72 performs a tap-detemninirig process similar to that by the tap-determining cir- 
cuit 7 shown in Fig. 1, whereby tap information on taps to be generated by the class-tap; generating circuit 64 and the 
prediction-tap generating circuit 65 is determined and supplied to the class-tap generating circuit 64 and the prediction- 
tap generating circuit 65, The tap-detemriining circuit 72 also calculates the predetemnined reference value which is 
used for the tap-determining process. 

35 [0081] Referring to the flowchart shown in Rg. 8, a learning jDrocess in which the learning apparatus shown in Rg. 
7 finds prediction coefficients for classes and the predetermined reference value is described below. 
[0082] Original images {moving picture) as teacher data are supplied to the learning apparatus in units of frames, 
and the images are sequentially stored in the frame memory 61 . the original images, stored in the frame memory 61 , 
are supplied to the blurring circuit 62, in which the supplied images are processed to be blurred images.. The . blurring 

40 circuit 62 can generate blurred images having different degrees of blurring in units of frames. 

[0083] The blurred images obtained in the blurririg circuit 62 are sequentially supplied as student data to the frame 
memory 63. . 

[0084] . When the blurred images con-esponding to all the original images prepared for the learning process are 
stored in the frarne memory 63, as described above, the tap-detemriining circuit 72 finds a reference value for the tap- 
45 determining process, as described later, and the process proceeds to step S22. 

[0085] In step S22, the tap-determining circuit 72 determines blurring pixels constituting the taps, similarly to the 
tap-determining circuit 72 shown in Rg. 1 , and outputs tap infomiation on the blurring pixels to the class-tap generating 
circuit 64 and the prediction-tap generating circuit 65. 

[0086] In step S23, in accordance with tap infomnation from the tap-detennining circuit 72, the class-tap generating 
50 circuit 64 and the prediction-tap generating circuit 65 read the blurring pixels constituting the class taps and the predic- 
tion taps from the frame memory 63, respectively, whereby the class taps and the prediction taps are formed. The dass 
taps are supplied to the classifying circuit 66, and the prediction taps are supplied to the adder circuit 67. 
[0087] In step S24, the classifying circuit 66 uses the class taps from the class-tap generating circuit 64 to classify 
the original pixels of interest, similarly to the classifying circuit 4 shown in Rg, 4. Class codes as the classifying results 
55 are supplied as addresses to the prediction-tap memory 68 and the teacher-data memory 70. 

[0088] The process proceeds to step S25, and the addition of prediction taps (student data) and the addition of 
teacher data are performed. 

[0089] In step S25, the prediction-tap memory 68 reads values stored at the addresses coi^responding to the class 
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codes output from the classifying circuit 66, and supplies the values to the adder circuit 67. The adder circuit 67 per- 
forms computation corresponding to summation (Z) as a multiplier for the prediction coefficient w in each left side of the 
normalization equations (7), using the stored values supplied from the prediction-tap memory 68 and the blumng pixels 
constituting the prediction taps, which are supplied from the prediction-tap generating circuit 65. The adder circuit 67 
5 overwrites the computation result at an address in the prediction-tap memory 68, which corresponds to the class code 
output by the classifying circuit 66. 

[0090] In step S25, the teacher-data memory 70 reads the values stored at the addresses corresponding to the 
class codes output from the classifying circuit 66, and supplies the read values to the adder circuit 69. The adder circuit 
69 reads the original pixels of interest from the frame memory 61 , and reads necessary blurring pixels from the frame 
10 memory 63. The adder circuit 69 uses the read pixels and the stored values supplied from the teacher-data memory 70 
to perform computation corresponding to summation (I) in each right side of the normalization equations (7). The adder 
circuit 69 stores the computation results at the addresses con-esponding to the class codes output from the classifying 
circuit 66 so that they are ovenwritten. . 

[0091] The process proceeds to step S26, and determines whether the above-described processing using all the 

15 original pixels in the present block of interest has been performed. If the process has detemiined negatively, the proc- 
ess proqeeds back to step S23. In step S23, among the original pixels in the block of interest, those unused as original 
pixels of interest are used as new original pixels of interest, and similar processing is performed. 
[0092] If, for example, a controller or the like has determined in step S26 that the above -described processing has 
been performed using all the original pixels in the block of interest as original pixels of interest, the process proceeds to 

20 step S27, and the controller or the like determines whether there is a block to process, in other words, whether blurring 
pixels corresponding to the next block to be processed are stored in the frame memory 63. If the controller or the like 
has determined in step S27 that the blumng pixels corresponding to the next block to be processed are stored in the 
frame memory 63, the process proceeds back to step S22. In step iS22, by using the stored block as a new block of 
interest, and using one of original pixels in the block of interest as a new original pixel of interest, similar processing is 

25 subsequently performed. 

[0093] If the controller or the like has determined in step S27 that the blurring pixels corresponding to the next block 
to be processed are not stored in the f rame memory 63, in other words, when the above-described processing is per- 
formed using all the original pixels prepared for learning, the process proceeds to step S28. In step S28, the computing 
circuit 71 sequentially reads the values stored at the addresses corresponding to the class codes from the predictipn- 

30 tap memory 68 and the teacher-data memory 70. and establishes the normalization equations (7), which use the read 
values. The computing circuit 71 finds prediction coefficients for the classes by solving the normalization equations (7). 
In step S29, the computing circuit 71 outputs the obtained prediction coefficients for the classes, and the learning proc- 
ess ends. 

[0094] In the above-described learning process of the prediction coefficients, a class may be generated in which 
35 normalization equations required for finding prediction coefficients cannot be obtained. For such a class, default predic- 
tion coefficients, or prediction coefficients obtained by predetermined learning with all pixels which are not classified, 
may be output, 

[0095] Referring to Fig. 9, the structure of the tap-deternaining circuit 72 (shown in Fig.. 7) is describe(^ belovy. 

[0096] As shown in Fig. 9, the tap-determining circuit 72 includes a reading unit 81 , a standard-deviation calculator 
40 82, a determining unit 83, and a reference-value generator 84. The reading unit 81, the standard-deviation calculator 

82, and the determining unit 83 are identical in structure to the reading unit 11, the standard-deviation calculator 12, 

and the determining unit 13 (shown in Fig. 5), respectively. Accordingly, the tap-detemiining circuit 72 is identical in 

structure to the tap-detemriining circuit 7 (shown in Fig. 5), except that the reference-value generator 84 is provided 

instead of the reference-value-setting unit 14 (shown in Fig. 5). 
45 [0097] Instep S21 shown in Fig. 8, the reference-value generator 84 finds a reference value forthe tap-determining 

process, and supplies the value to the determining unit 83. 

[0098] In steps S31 to S37 shown in Rg. 10, the tap-determining circuit 72 having the above-described structure 
performs processing similar to that in steps S1 1 to SI 7, whereby tap information. for each block is detemnined. 
[0099] With reference to Figs. 1 1 and 12, calculation of the reference value by the reference- value generator 84 

50 (shown in Rg. 9) is described below. 

[0100] The reference-value generator 84 reads, from the frame memory 61. all original images prepared for the 
learning process, and reads, from the frame memory 63, blun-ed images (student data) in which the original images are 
blurred. The reference-value generator 84 divides each of the original images into blocks as described above, gener- 
ates, for pixels in each of the blocks, taps having different tap widths, and finds relationships between the tap widths and 

55 the average of standard deviations of the blurred images coristituting the taps. 

[0101] In other words, when an attention is paid to acertain block, the reference-value.generator 84 uses each orig- 
inal pixel of the block as an original pixel of interest to generate taps having a tap width ]. The reference-value generator 
84 finds standard deviations of taps {generated for each original pixel, and calculates the average of the standard devi- 
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ations in the block. Specifically, when the total number of original pixels in a block #i is represented by M, the total 
number of blurring pixels constituting taps is represented by K. the pixel level of the k-th blurring pixel in taps having a 
tap width j is represented by ^ r. in which the k-th blurring pixel corresponds to the m-th pixel in the block #i, and the 
average of taps having the tap width j (i.e., the average of bluning pixels constituting the taps) is represented by 
5 meanmj. score(i.j) is computed in accordance with expression (8) by the reference-value generator 84. 

[0102] The reference-value generator 84 generates taps for each original pixel in the block #1, while changing the 
tap width j into several values, and sequentially the average. score{i,i), of the standard deviations of the taps having 
each value of the tap width j. 

[0103] - The reference-value generator 84 similarly computes the average, score(i.j). of the standard deviations of 

70 the taps having each value of the tap width i. 

[0104] As described above, for each block. #1 . the reference-value generator 84 finds relationships between the tap 
width j and the average, score(i,j), of the standard deviations of the taps having the tap width j, as shown in Fig. 1 1 . 
[0105] ng. 1 1 shows relationships between the tap width j and the average. score(i,j). of the standard deviations of 
the taps having the tap width j in four blocks #1 to #4. In Fig. 11 . the circular symbols indicate values concerning the 

IS block #1 . the cross symbols indicate values concerning the block #2. the triangular symbols indicate values conceming 
the block #3, and the square symbols indicate value concerning the block #4. 

[0106] Rg. 11 shows values of the average, score(i,j), of the standard deviations of the taps, which are obtained 
when the tap width j is set to zero (0) to 1 0. 

[0107] The average, score(i.j), of the standard deviations of taps generated for each original pixel of the block i indi- 
20 cates how the pixel level of a blurred image for the block #i changes, that is. the degree of blurring. Accordingly, as the 
greater (smaller) the score(i,j), the smaller (greater) the degree of blurring. 

[0108] When finding the average, score(i,j). of the standard deviations of taps, the reference-value generator 84 
simultaneously finds predicted values based on the original pixels by performing classification adaptive processing 
using the taps having the tap width ]. , 

25 [0109] In other words, the reference-value generator 84 uses the original pixels in the block #i and the blurring pix- 
els to fix the tap width j to a predetermined value, establishes, and solves the normalization equations (7). whereby pre- 
diction coefficients for the classes are found. The reference-value generator 84 also uses the predtotion coefficients for 
the classes, fixes the tap width j to the predetemriined value, and performs computation based on expression (1). 
whereby predicted values based on each original image in the block #i are found. 

30 [0110] The reference-value generator 84 performs similar processing while sequentially changing the tap width j. 
and performs similar processing on other blocks. Thereby, for the original pixels in each block, predicted yalues e(i,j) 
obtained when taps having each value of the tap width j are found. In other words, for original pixels in each block #i, 
the relationship between the tap width j of taps for predicting the original pixels and a predicted value (i.j) based on each 
original pixel is found. 

35 [0111] The reference-value generator 84 detemriines how much the obtained predicted value e(i,j) is close to the 
original image. In other words, for, e.g., the block #i, the reference -value generator 84 finds, as a signal-to-noise ratio of 
the predicted value e(i.j), the reciprocal of the sum of absolute-value differences between the original pixels and each 
predicted value e(ij) obtained with taps having the tap width j. Si§nal-to-noise ratios obtained for the four blocks #1 to 
#4 shown in Fig. 1 1 are shown in Fig. 12. . 

40 [0112] In Fig. 12, a predicted value e(1.0), obtained using taps having a tap width of zero, has a most preferable 
signal-to-noise ratio, which means that the e(1 ,0) is closest to the original image. In the reference-value generator 84. 
a block #i from which the predicted value e(i.j) closest to the original image is obtained, and a tap width j are detected. 
The block #1 and the tap width j, detected as described above, are hereinafter referred to as the "optimal. block and the 
"optimal tap width", respectively. 

45 [0113] Since each predicted value e(i,j). obtained when taps having an optimal tap width j are generated for the 
optimal block, is closest to the original image, a standard deviation (hereinafter refen-ed to also as an "optimal standard 
deviation") of taps when the taps are generated to have an optimal tap width is considered as an optimal standard devi- 
ation for taps generated for other blocks. 

[01 14] In other words, it is possible from a statistic aspect that, for each of the other blocks, when taps having a tap 
50 Width, in which a standard deviation from the block is used as the optimal standard deviation, are generated, predated 
values closest to the original image are obtained. 

[0115] Accordingly, the reference-value generator 84 finds the optimal standard deviation, and supplies the optimal 
standard deviation as a predetermined reference Value to the determining unit 83. 

[0116] 1n Fig. 12, the predicted value e(1,0). obtained for the block #1 by using taps having a tap width j of zero, has 
55 a most preferable signal-to-noise ratio. Thus! among values of the score(1 .j) obtained for the block #1 , which are indi- 
cated by the circular symbols shown in Rg. 11. the value 20, obtained when .the tap widfri j is zero, is found as the opti- 
mal standard deviation. 

[0117] For the other blocks, each value indicated by the bold an-ow shown in Fig. 1 1 is the tap width j based on 
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which a standard deviation corresponding to the optimal standard deviation is obtained. Therefore, in the tap-determin- 
ing processes shown Figs. 6 and 1 0, the determined taps have the tap width j based on which the optimal standard 
deviation is obtained. 

[0118] Although the tap width j takes integer values, the tap width (each value indicated by the bold arrow shown in 

5 Fig. 11) based on which a standard deviation corresponding to the optimal standard deviation is obtained does not 
always take integer values (does not take integer values in many cases), as can be clear from Fig: 1 1 . Accordingly, in 
the tap-determining processes shown in Figs. 6 and 10, when the tap width based on which a standard deviation cor- 
responding to the optimal standard deviation is obtained is not an integer value, an integer value closest the tap width 
which is not the integer value is detemiined as the final tap width. 

10 [0119] Specifically, in the tap-determining process shown in Fig. 6 (similarly to that shown in Rg. 10), the process 
determines whether the score(i,j) from the standard-deviation calculator 12 is close to the reference value set in the ref- ' 
erence-value setting unit 14 (step SI 5). If the process has detemiined affirmatively, a tap width ] at the time the deter- 
mination is performed is output as tap information (step SI 7). That "the score(i,j) is close to the reference value" means 
that a standard deviation score(i,j) (the average of standard deviations obtained for the block in this embodiment) of taps 

15 having a tap width j of an integer value, obtained when the taps are generated, is closest to the reference value. 

[0120] In the tap-detemriining processes shown in Figs. 6 and 1 0. it is possible to perform finding a tap width based 
on which a standard deviation corresponding to the optimal standard deviation is obtained, and generating, when the 
tap width is not an integer value, taps in which the average of tap widths is used as the average of the tap widths. In 
other words, when the tap width based on which the standard deviation corresponding to the optimal standard deviation 

20 is, for example, 1 .5, by generating taps so that the tap width between pixels is 2, the average of tap widths can be set 
to 1.5. 

[0121] As described above, the learning process and the blur-reducing process are performed by finding an optimal 
standard deviation, and generating taps having a tap width based on which a standard deviation corresponding to the- 
optimal standard deviation is obtained. Therefore, the processing perfomnance can be increased. 

25 [0122] In other words, in the learning process, classification adapted for the degree of blurring of each image is per- 
formed, and prediction taps adapted for the degrees of blurring are generated. As a result, prediction coefficients 
adapted for the degrees of blurring of images, which correspond to classes, can be obtained. Also in the blur-reducing 
process, classification adapted for the degree of blurring of each image is performed, prediction taps adapted for the 
degrees of blurring are generated, and predicted values based on original pixels are found using the prediction coeffi- 

30 cients for classes, which are adapted for the degrees of blurring. As a result, clear images (Images in which blurs are 
reduced) can be obtained, regardless of the degrees of blurring. 

[0123] Specifically, when the classification adaptive processing is performed using, for example, input images hav- 
ing a small degree of blurring, as described above, classification in which the characteristics of original pixels of interest 
are reflected can be performed by preferably using input pixels which are relatively close to the origirial pixels of interest. 
35 When the classification is performed using input images having a large degree of blurring, classification in which the 
characteristics of original pixels of interest are reflected can be performed by preferably using input pixels which are rel- 
. atively far from the input pixels. 

[0124] In the case shown in Fig. 11, the greater the s6ore(i,j), the smaller the degree of blurring, as described 
above. Thus, the degree of blumng increases in the order of the blocks #1, #2, #3, and #4. In the case shown in Fig. 
40 11. taps that have tap widths of 0, 1 . 3, and 8 are generated for the blocks #1 , #2, #3, and #4. 

[0125] Accordingly, according to the present invention, as the degree of blurring increases, class taps are gener- 
ated using input pixels which are far from original pixels of interest. Thus, classification in which the characteristics of 
images are reflected is performed. 

[0126] In the first embodiment, tap information is determined based on a standard deviation of taps. However, the 
45 tap Infonnation may be detemnined based on a statistic other than the standard deviation of taps. That is, the.tap infor- 
mation may be determined based on, for example, distribution of pixels constituting the taps, the absolute sum of differ- 
ences among the pixels, the absolute sum of second-order differences (the absolute sum of differences among 
differences of pixels), or the like. 

[0127] In the first €mt>odiment, a tap width is changed for each block composed of a plurality of pixels. However, the 

50 tap width may be changed for, for example, each pixel unit. 

. [0128] In the first embodiment, prediction coefficients are prestored in the coefficiisnt RAM 5. However, the predic- 
tion coefficients may be supplied to the image processing apparatus, with, for example, input images. Similariy, a refer-, 
ence value does not need to be set In the reference-value setting unit 14, but may be supplied to the image processing 
. apparatus, with the input images. 

55 [0129] In the first embodiment, the arrangement of taps (pixels constituting taps) is changed by changing the tap 
width based on a statistical value of input images. However, the tap structure may be changed by,' for example, changing 
the positions of pixels constituting the tap. Tap widths do not need to be at equal intervals. For example, it is possible to 
radially expand four taps in the four corners of, for example, a 3 x 3 tap, using the center pixel as the center. 
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[0130] In the first embodiment, a first-order expression is used to find predicted values based on original pixels. 
However, an expression having an order not less than the second order may be used to find the predicted values. 
[0131] In the learning process, addition corresponding to the summation (Z) in expressions (7) is performed using 
prediction taps, as described above. However, addition using prediction taps having different tap widths is perfomried 
using con-esponding pixels in the prediction taps. 

[0132] VVhen a prediction tap is generated so as to have nine pixels (three pixels by three pixels) around the center 
pixel in the center, in the first embodiment, a prediction tap having a tap width of zero (Fig. 3A) and a prediction tap hav- 
ing a tap width of one (Fig. 3B) are generated. In this case, a pixel of the prediction tap having a tap width of zero, for 
example, the top leftest pixel is added to the top leftest pixel of the prediction tap having a tap width of one. 
[0133] ' Also classification of class taps having different tap widths is performed using a similar technique. Therefore, 
by way of example, when the pixel level of each pixel constituting the prediction tap (shown in Fig. 3A) having a tap width 
of zero is equal to the pixel level of each pixel constituting the prediction tap (shown in Fig. SB) having a tap width of 
one. a result of classification using the class tap shown in Rg. 3A and a result of classification using the class tap shown 
in Fig. 38 are the same (the taps are classified into the same class). 

[0134] Next, with reference to Fig. 1 , a second embodiment of an image processing apparatus to which the present 

invention is applied is described below. 

[0135] The image processing apparatus according to the second embodiment differs from the first embodiment in 
that, when an SD image, having a low resolution is input as an input image to the image processing apparatus, the 
image processing apparatus uses the classiricatton adaptive processing to process the input image, whereby an HD 
image having a high resolution is output. Since the components of the image processing apparatus according to the 
second embodiment are identical to those in the first embodiment, descriptions thereof are partially omitted. 
[0136] The tap-determining circuit 7 detennines, based on waveform characteristics of the input images stored in 
the frame memory 1, input pixels constituting the class taps and the prediction taps, and supplies the class-tap gener- 
ating circuit 2 and the prediction-tap generating circuit 3 with information on the input pixels constituting the class taps 
and the prediction taps. 

[0137] The tap-determining circuit 7 determines, as basic taps, for example, a square class tap to be generated by 
the class-tap generating circuit 2 and a square prediction tap to be generated by the prediction-tap generating circuit 3. 
each of which consists of five vertical pixels and five horizontal pixels, in which an input pixel adjacent to a pixel of inter- 
est is used as a center pixel (meaning, for example, a pixel at the position of the centroid). as indicated by the dotted 
line shown in Fig. 1 3. 

[0138] In Fig. 13, the cross symbols indicate pixels (HD pixels) constituting an HD image, and the circular symbols 
indicate pixels (SD pixels) constituting an SD image as an input image. Accordingly, in Fig. 13, the SD image has a size 
obtained by reducing the numbers of vertical pixels and horizontal pixels to half. 

[0139] When determining each basic tap, the tap-detemiiriing circuit 7 detects directions from the center pixel as a 
start point to other SD pixels (input pixels) of the basic tap, as shown in, for example, Fig. 14. When the basic tap con- 
sists of 5 X 5 pixels, sixteen directions to Die ^rom the center pixel as the start point are detected, as shown in Fig. 
14. V 

[0140] For each direction Dj (i = 1 , 2. .... 1 6). the tap-detennining circuit 7 detennines, based on waveform charac- 
teristics (pixel-level characteristics) of SD pixels in the direction, SD pixels for finally generating a tap. In other words, 
for each direction Dj, based on the wavefomi characteristics of SD pixels in the direction, the positions of the SD pixels 
constituting a tap in the direction are shifted. The tap-determining circuit 7 supplies the class-tap generating circuit 2 
and the prediction-tap generating circuit 3 with, as tap information, information on the positions of SD pixels for finally 
generating the tap as a result of the shifting. 

[0141] With reference to the flowchart shown in Fig. 16, a resolution-increasing process that increases the resolu; 
tion of an input image by the image processing apparatus shown in Fig. 1 is described below. 

[0142] SD images (moving picture) to be processed by the resolution -increasing process are sequentially supplied 
as input images in units of frames to the frame, memory 1 . In the frame memory i, the input images are sequentially 
stored in units of frames. 

[0143] A predetennined HD pixel based on which a predetermined is found is used as a pixel of interest Iri step 
S41. the tap-determining circuit 7 determines, based on waveform characteristics of the input images stored in the . 
frame memory 1 , input pixels for generating taps based on the pixel of interest, and outputs tap information on the posi- 
tions of the input pixels to the class-tap generating circuit 2 and the prediction-tap generating circuit 3. 
[0144] In step S42, when receiving the tap information from the tap-detemnining circuit 7. the class-tap generating 
circuit 2 and the prediction-tap generating circuit 3 respectively read, from the frame memory 1 , the input pixels for gen- 
erating class taps arid prediction taps based on the pixel of interest from which a predicted value is found, and generate 
the class taps and the'prediction taps. The class taps and the prediction taps are supplied to the. classifying circuit 4 
and the prediction arithmetic circuit 6. respectively. 

[0145] In step S43, when receiving the class taps from the class-tap generating circuit 2, tiie classifying circuit 4 
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classifies pixels of interest, based on the class taps, and outputs, addresses, class codes obtained as a result of the 
classification to the coefficient RAM 5. In step S44, the coefficient RAM 5 reads a prediction coefficient stored at the 
address con-esponding to each class code, and supplies the class code to the prediction arithmetic circuit 6. 
[0146] In step S45, the prediction arithmetic circuit 6 uses the prediction tap from the prediction-tap generating cir- 
5 cuit 3 and the prediction coefficient from the coefficient RAM 5 to perform linear predicting computation represented by 
expression (1), whereby the predicted values E[y] based on the pixels y of interest are found. After that, the process 
proceeds to step S46. In step S46, the prediction arithmetic circuit 6 outputs, as HD pixels having increased resolution, 
to the predicted values E[y] based on the pixels y of interest, which are obtained in step S45, and the process proceeds 
to step S47. 

10 [0147] In step S47, the controller or the like determines whether all predicted values of HD pixels constituting the 
frame including the pixel of interest have been found. If the controller or the like has determined negatively, the process 
proceeds back to step S41 , and the steps from step S41 are repeatedly perfomned using, as new pixels of interest. HD 
pixels from which predicted values have not been found yet. 

[0148] If the controller or the like has determined in step S47 that all the predicted values of HD pixels constituting 
IS the frame including the pixel of interest have been found, the process proceeds.to step S48. In step S48, the controller 
or the like determines whether an input image (SD image) corresponding to the next frame of an HD image to be proc- 
essed is stored in the frame memory 1 . If the controller or the like has determined in step S48 that the input image cor- 
responding to the next frame of the HD image to be processed is stored in the frame memory 1 , a predetermined HD 
pixel in the next frame of the HD image to be processed is used as a new pixel of interest, and the process proceeds 
20 back to step Si . Subsequently, similar processing-is repeatedly perfomned. 

[0149] If the controller or the like has determined in step 848 that the input image con-esponding to the next frame 
of the HD image to be processed is not stored in the frame memory 1 , the resolution-increasing process is tenminated. 
[0150] With reference to the flowchart shown in Fig. 17, a first modification of the tap-determining process per- 
formed by the tap-determining circuit 7 in step S41 shown in Fig. 1 6 is described below. 
25 [0151] In addition to an entire set of SD pixels for use in classification and computation based on the linear predic- 
tion expressions (1 ), each of the pixels is hereinafter refen-ed to also as a "tap". 

[0152] In step S51 . the tap-detentiining circuit 7 sets a basic tap. Specifically, in step S51 , the tap-detemiining cir- 
cuit 7 sets, as a basic tap, a tap composed of five vertical pixels and five horizontal pixels in which an input pixel at the 
position of a pixel of interest is used as a center pixel, as indicated by the dotted line shown in Fig. 13. The tap-deter- 
30 mining circuit 7 proceeds to step S52. 

[01 53] In step S52, the tap-determining circuit 7 detects sixteen directions D^ to D^e from the center pixel as a start 
point to other SD pixels in the basic tap. and the process proceeds to step S53. 

[01 54] In step S53. a variable i that represents the sixteen directions detected in step S52 is initialized to, for exam- 
ple. "1" by the tap-determining circuit 7, and the tap-detemnining circuit 7 proceeds to step S54. In step S54. a variable 
35 j that represents a tap in each direction D, is initialized to, for example, "1 " by the tap-detemnining circuit 7, and the tap- 
determining circuit 7 proceeds to step S55. 

[0155] In step 855, the tap-determining circuit 7 determines whether the pixel level of an SD pixel, used as the ]-th 
tap Tj I from the center pixel in the direction Dj, is extreme. 

[0156] If the tap-determining circuit 7 has determined in step 855 that the level of the SD pixel, used as the j-th tap 
40 Tjj from the center pixel in the direction Dj, is not extreme, the tap-detemriining circuit 7 proceeds to step S56, and the 
tap T|j in the direction Dj and outer taps therefrom (far from the center pixel) are moved, for example, one pixel away 
from the center pixel. The tap-determining circuit 7 proceeds back to step 855, and similar processing is repeatedly per- 
formed. 

[0157] If the tap-determining circuit 7 has detemnined in step S55 that the level of the SD pixel, used as the j-th tap 
45 Tj J from the center pixel in the direction D|, is extreme, the tap-determining circuit 7 proceeds to step 857. In step 857. 
the variable ] is incremented by one, arid the tap-determining circuit 7 proceeds to step 858. In step 858, the tap-deter- . 
mining circuit 7 determines whether the variable j is not more than the number Jj of pixels (in the direction D|) constitut- . 
ing taps other than the center pixel. 

[0158] Here, in Fig. 14, Jiforthe directions Di, D3, D5 D15 is, for example, 2, and Jj for the directions Dg, D4. Oq, 

50 Die is, for example, 1. 

[0159] If the tap-determining circuit 7 has detemiined in step 858 that the variable j is hot more than the number J| 
of pixels (in the direction D|) constituting taps other than the center pixel, that is. when all SD pixels (in the direction Dj) 
constituting the taps other than the center pixel are not extreme, the tap-detemnining circuit 7 proceeds back to step 
855, and similar processing oh taps Tjj at the positions of the SD pixels which have not reached their extremes is 

55 repeatedly performed. 

[0160] If the tap-determining circuit 7 has determined in step 858 that the variable j is more than the number Jj of 
pixels (in the direction Dj) constituting taps other than the center pixel, that is, when all SD. pixels (ip the direction D|) 
constituting the taps other than the center pixel are extreme, the tap-determining circuit 7 proceeds to step S59, and 
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the variable i is incremented by one. The tap-determining circuit 7 proceeds to step S60, and determines whether the 
variable i is not more than a total number I (16 in the case shown in Fig. 14) of the directions detected in step S52, 
[0161] If the tap-determining circuit 7 has determined in step S60 that the variable i is not more than the total 
number I, that is, when the SD pixels which are extreme have not been used as taps yet for all the directions detected 

5 in step S52. the tap-determining circuit 7 proceeds back to step S54, and similar processing is repeatedly performed. 
[0162] If the tap-determining circuit 7 has determined in step S60 that the variable i is more than the total number 
I. that is, when the SD pixels which are extreme have been used as taps for all the directions detected in step S52, the 
tap-determining circuit 7 proceeds to step S61 . In step S61 , the information on the positions of the SD pixels constitut- 
ing the taps are output as tap information to the class-tap. generating circuit 2 and the prediction-tap generating circuit 

10 3, and the process returns. 

[0163] According to the first embodiment of the above-described tap-determining process, the class-tap generating 
circuit 2 and the prediction -tap generating circuit 3 generates taps as shown in Figs, 18A and 1 88. 
[0164] In other words, Figs. 18A and 18B, Rgs. 20A and 20B. and Figs. 22A and 22B show the direction Di, to 
which attention is paid among the sixteen directions shown in Fig. 14, and SD pixels in the direction Dg which is 180- 

15 degree opposite to the direction D^. 

[0165] In Figs. 1 8A and 1 88, Figs. 20A and 208, and Figs. 22A and 228, the position of a center pixel for the tap is 
represented by "n", the position of a pixel which is m pixels (where m is a positive integer) away from the position n in 
the direction D^ is represented by "n-m", and the position of a pixel which is m pixels away from the position n in the 
direction Dg is represented by "n+m". 

20 [0166] In Figs. 18A and 188, Figs. 20A and 208, and Figs. 22A and 228, each cross symbol indicates (the pixel 
level) of an SD pixel, and each circular symbol indicates (an SD pixel as) a tap. 

[0167] Fig. 18A shows basic taps, in which two adjacent SD pixels in the direction D^ with respect to an SD pixel 
as the center pixel, and two adjacent SD pixels in the direction Dg are used as taps. 

[0168] Fig. 1 88 shows taps generated from the basic taps shown in Fig. 1 8A by the tap-detemiining process shown 
25 in Fig. 1 7. In Fig. 1 88, with respect to an SD pixel used as the center pixel, an SD pixel having a maximum and an SD 
pixel having a minimum, which first appear in the direction D^, are used as taps, and an SD pixel having a minimum and 
an SD pixel having a maximum, which first appear in the direction Dg. are used as taps. 

[0169] With reference to the flowchart shown in Fig. 19, a second modification of the tap-determining process in 
step S41 (shown in Fig, 1 6) by the tap-determining circuit 7 is described below. 
30 [0170] In steps S71 and S72, the tap-determining circuit 7 performs processing similar to that perfonned in steps 
S51 and S52 shown in Fig. 1 7, thereby setting the basic taps and detecting the sixteen directions Di to D^e to other SD 
pixels in each basic tap in which the center pixel is used as a start point. 

[0171] The process proceeds to step S73, the tap-determining circuit 7 initializes a direction-representing variable 
i detected in step S72 into, for example, "1", and the process proceeds to step S74. In step S74, a dynamic range DRi 
35 of SD pixels in the direction Dj used as taps is computed. In other words, in step S74, the tap-determining circuit 74 
detects a maximum and a minimum from among all the SD pixels (including the center pixel) used as taps in the direc- 
tion Dj, and finds the difference between the maximum and the minimum as the dynamic range DRj. 
[0172] After finding the dynamic range DRj, the tap-determ\ning circuit 7 determines in step S75 whether the 
dynamic range DRj is greater than a preset threshold "th^ If the tap-determining circuit 7 has determined in step S75 

40 that the dynamic range DRj is not more than the threshold th, that Is, when it is found that the taps in the direction D| are 
constant since there is no large charige among the SD pixels as the taps in the direction D|, the process proceeds to 
step S76. In step S76. the SD pixels excluding the center pixel, which constitute the taps, are changed so that intervals 
among the taps in the direction D| are increased by one pixel, and the process proceeds to step S74. Subsequently, 
similar processing is repeatedly perfomried. 

45 [0173] If the tap-determining circuit 7 has determined in step S75 whether the dynamic range DRj is greater than 
the threshold th. that is, when it is found that the taps in the direction Dj are not constant since there is a relatively large 
change among the SD pixels as the taps, the process proceieds to step S77. In step S77, the variable i is incremented 
by one. The process proceeds to step S78, and the tap-determining circuit 7 detemnines whether the variable i is not 
more than the total number I of directions detected in step S72. 

50 [0174] If the tap-determining circuit 7 has determined in step S78 that the variable i Is not more than the total 
number I. that is, when SD pixels in which the dynamic range is greater than the threshold th have not been used as tap 
yet, this process proceeds back to step S74. Subsequently, similar processing is repeatedly performed. 
[0175] If the tap-detemnining circuit 7 has determined in step S78 that the variable i is mbre than the total number 
I, that is, when SD pixels in which the dynamic range is greater than the threshold th are used as taps in all the direc- 

55 tions detected in step S72. the process proceeds to step S79. In step S79, the tap-detemiining circuit 7 outputs, as tap 
infonnation, infomnation oh the positions of the SD pixels constituting the taps to the class-tap generating circuit 2 and 
the prediction-tap generating circuit 3, and the process returns. 

[0176] According to the above-described second modification of the tap-detemiining process, the class-tap gener- 
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ating circuit 2 and the prediction-tap generating circuit 3 generate, for example, taps as shown in Figs. 20A and 20B. 
[01771 Fig. 20A shows basic taps similarly to Fig. 1 8A, in which, with respect to an SD pixel as the center pixel, two 
adjacent SD pixels in the direction D^, and two adjacent SD pixels in the direction Dg are used as taps. 
[0178] Fig. 20B shows taps generated from the basic taps by the tap-determining process shown in Fig. 19. In all 
5 the directions to Dg, SD pixels in which the dynamic ranges DRi to DRg are greater than the thrieshold th are used 
as taps. 

[0179] In the cases shown in Fig. 19 and Figs. 20A and 20B, intervals among the SD pixels as the taps are 
expanded so that the dynamic range DRj, which is the difference between the maximum and the minimum among the 
SD pixels as the taps in the direction D^^, is greater than the threshold th. In addition, by way of example, the intervals 
10 among the SD pixels as the taps may be expanded so that the difference between two adjacent SD pixels as the taps 
in the direction Dj is greater than the threshold th. 

[0180] With reference to the flowchart shown in Fig. 21, a third modification of the tap-determining process per- 
formed in step 841 (shown in Rg. 16) is described below. 

[0181] In steps S81 to S83, the tap-determining circuit 7 performs processing similar to that performed in steps S51 
15 to S53 shown in Fig. 17. In other words, the tap-determining circuit 7 sets basic taps, detects the sixteen directions D^ 
to Die to other SD pixels in the basic taps in which the center pixel is used as the a start point, and initializes variable i. 
[0182] In step S84, the tap-detemriining circuit 7 finds, for example, a line (hereinafter referred to as also an 
"approximate line") Lj, which is a function approximating (the pixel levels of) SD pixels used as basic taps in the direction 
D|, and proceeds to step S85. In step S86, the tap-determining circuit 7 finds errors (e.g., squared errors) of the SD pix- 
20 els (including the center pixel) as the taps in the direction Dj with respect to the approximate line Lj, and finds an accu- 
mulated value of the en-ors. In step S86, the tap-determining circuit 7 detemnines whether the accumulated value of the 
errors is greater than a predetermined threshold. 

[0183] If the tap-determining circuit 7 has determined in step S86 that the accumulated value of the errors is not, 
greater than the predetennined threshold, that is, when it is found that the SD pixels as the taps in the direction D| are 

25 constant since there is no particular change among the SD pixels, the tap-determining circuit 7 proceeds to step S87. 
In step S87, all the taps excluding the center pixel in the direction D| are moved along the direction D-, (outward) by, for 
example, one pixel, and proceeds back to step S85, Subsequently, similar processing is repeatedly performed. 
[0184] If the tap-determining circuit 7 has determined in step S86 that the accumulated value of the errors is greater 
than the predetermined threshold, that is, when it is found that the SD pixels as the taps in the direction Dj are not con- 

30 stant since there is relatively large change among the SD pixels as the taps, the tap-determining circuit 7 proceeds to 
step S88, and the variable i is incremented by one. In step S89, the tap-detemnining circuit 7 determines whether the 
variable i is not more than the total number I of the directions detected in step S82. 

[0185] If the tap-determining circuit 7 has determined in step S89 that the variable i is not more than the total 
number I, that is, when SD pixels in which the accumulated value of the errors is greater than the predetermined thresh- 
35 old value have not yet been used as taps in all the directions detected in step S82, the tap-determining circuit 7 pro- 
ceeds back to step S84. Subsequently, similar processing is. repeatedly performed. . 

[0186] If the tap-determining circuit 7 has determined in step S89 that the variable I is more than the total number 
I, that is, when the SD pixels in which the accumulated valu^ of the errors is greater than the predetermined threshold 
value have been used as taps, the tap-determining circuit 7 proceeds to step S90. In step S90, infomriation on the posi- 
40 tions of the SD pixels constituting the taps is output as tap infonnation to the class-tap generating circuit 2 and the pre- 
diction-tap generating circuit 3, and the process returns. 

[0187] According to the above-described third modification of the tap-determining process, the class-tap generating 
circuit 2 and the prediction-tap generating circuit 3 generate taps as shown in Figs. 22A and 228. 
[0188] Fig. 22A shows basic taps similarly to Fig. 1 8A, in which, with respect to an SD pixel as the center pixel, two 
45 adjacent SD pixels in the direction D^ , and two adjacent SD pixels in the direction Dg are used as taps. 

[01 89] Fig. 228 shows taps generated from the basic taps by the tap-determining process shown in Fig. 21 . In all 
the directions D^ to Dg, SD pixels in which an accumulated value of errors f rom approximate lines to Lg is greater 
than a predeternriined value are used as taps. 

[0190] In the cases shown in Fig. 21 and Figs. 22A and 228, the SD pixels are generated based on the accumu- 
50 lated value of the errors of the SD pixels constituting the taps with respect to the line Lj approximating the SD pixels as 
the basic taps. In addition, the SD pixels may be generated based on, for example, distribution of the errors. 
[0191] In the case shown in Fig. 21 , in step S87, all the taps excluding the center pixel are moved along the direcr 
tion Dj by, for example, one pixel. However, the taps may be moved so that each interval among the taps in the direction 
Dj can be increased by, for example, one pixel. 
55 [0192] In the cases shown in Fig. 21 and Rgs. 22A and 228. a line is found as a functipn approximating the SD pix- 
els used as the basic taps in the direction D|. In addition, by finding a plane or a curved surface as a function approxi- 
mating SD pixels as the basic taps, which are positioned in at least two directions, the taps may be determined based . 
on errors with respect to the plane and curved surface. 
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[0193] \Amh reference to Fig. 23. a second type of the learning apparatus that finds the prediction coefficients for 
classes to be stored in the coefficient RAM 5 (shown in Fig. 1) is described below. 

[0194] The components of the second type, which are identical to those shown in Fig. 7, are denoted by identical 

reference numerals. Accordingly, the same description is omitted. 
5 [0195] Original images (high-resolution HD images), which are used as teacher data y, are supplied to a frame 

memory 61 in. for example, units of frames, and the frame memory 61 temporarily stores the supplied original images. 

A decimating circuit 91 read each original image stored in the frame memory 61 , which is used as the teacher data y in 

the learning of prediction coefficients, and decimates pixels constituting the original image, whereby a low-resolution 

SD image is generated. The decimating circuit 91 generates the SD image (indicated by the circular symbols shown in 
10 Rg. 13) by reducing the numbers of horizontal and vertical pixels (indicated by the cross symbols shown in Fig. 13) to 

half. The SD image is supplied to the frame memory 63. 

[0196] With reference to the flowchart shown in Fig. 24, a learning process for finding prediction coefficients for 
classes by the learning apparatus shown in Fig. 23 is described below. 

[0197] HD images as original images to be used as teacher data are supplied to the learning apparatus in units of 
t5 frames, and the supplied images are sequentially stored in a frame memory 61 . In step S91 , the HD image (as original 

images) stored in the frame memory 61 is supplied to a decimating circuit 91 . Each supplied image is converted into an 

SD image. From the decimating circuit 91 , the SD image is supplied and stored as student data in a frame memory 63. 

[0198] A tap-detennining circuit 72 uses, as pixels of interest, predetermined HD pixels in a frame of the SD image 

stored in the frame memory 63. In step S92, the tap-detemiining circuit 72 determines SD pixels for generating taps 
20 based on the pixel of interests, similarly to the tap-detemiining circuit 7 shown in Fig. 1 . The tap-detemnining circuit 72 

outputs tap information on the SD pixels to a class-tap generating circuit 64 and the a prediction-tap generating circuit 

65. 

[0199] In step S93, the SD pixels for generating class taps and prediction taps based on the pixels of interest are 
read from the tap-determining circuit 72 in accordance with the tap infomnation from the tap-determining circuit 72, and 
25 the class tap and the prediction tap are generated. The class taps are supplied to a classifying circuit 66, and the pre- 
diction taps are supplied to adder circuits 67 and 69. 

[0200] In step S94, the classifying circuit 66 uses the class tap from the class-tap generating circuit 64 to classify 
the pixels of interest, and supplies class codes, which are the classification results, as addresses to the a prediction-tap 
memory 68 and a teacher-data memory 70. 

30 [0201] In step S95, addition to prediction taps (student data) and teacher data is perfomned. 

[0202] In other words, in step S95, the prediction-tap memory 68 reads the stored address corresponding to the 
class code corresponding to the classifying circuit 66, and supplies the read address to the adder circuit 68. Using the 
stored values supplied from the prediction-tap memory 68, and the SD pixels for generating the prediction taps, which 
are supplied from the prediction-tap generating circuit 65, the adder circuit 67 performs computation conresponding to 

35 the summation (L) as a multiplier for the prediction coefficient w in each left side of the nomnalization equations (7). The 
adder circuit 67 ovenwrites the computation result at an address in the prediction-tap memory 68, which con^esponds to 
the class code output by the classifying circuit 66. 

[0203] Also in step S95. the teacher-data memory 70 reads \he stored address corresponding to the class code 
output from the classifying circuit 66, and supplies the read address to the adder circuit 69. The adder circuit 69 reads 

40 the pixels of interest from the frame memory 61 , and performs computation con-esponding to summation (E) in each 
right side of the normalization equations (7), using HD pixels used as the read pixels of interest, the prediction taps, and 
the stored value supplied from the teacher-data memory 70. The adder circuit 69 ovenwrites the computation result at 
an address in the teacher-data memory 70, which corresponds to the class code output from the classifying circuit 66. 
[0204] In step S96, a controller or the like determines whether the above-described processing has been perfomned 

45 using, as pixels of interest, all HD pixels constituting the frame including the pixel of interest. If the controller or the like 
has determined negatively, the process proceeds back to step S92. Subsequently, similar processing is repeatedly per- 
formed using, as new pixels of interest, HD pixels that have not been used as pixels of Interest yet. 
. [0205] If the controller or the like has detenmined in step S96 that the above -described processing has been per- 
formed using all the pixels constituting the frame including the pixel of interest, as further pixels of interest, the controller 

50 or the like proceeds to step S97. and determines whether the next frame to be processed is stored in the frame memory 
61. If the controller or the like has detemiined in step S97 that the next frame to be processed is stored in the frame 
memory 61, the process proceeds back to step.S91, and similar processing on the next frame to be processed is 
repeatedly performed. 

[0206] If the controller or the like has determined in step S97 that the frame to be processed is not stored in the 
55 frame memory 61 , that is, when the above-described processing has been performed using all the HD images prepared 
for learning, the process proceeds to step S98. In step S98, a computing circuit 71 sequentially reads values stored at 
addresses corresponding to the class codes in the prediction-tap memory 68 and the teacher-data memory 70, and 
solves the normalization equations (7), which use the read >reilues. whereby prediction coefficients for classes are 
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found. In step S99, the computing circuit 71 outputs the prediction coefficients for classes, and the learning process 
ends. 

[0207] In the above-described learning process of the prediction coefficients, a class nnay be generated in which 
normalization equations required for finding prediction coefficients cannot be obtained. For such a class, default predic- 
5 tion coefficients, or prediction coefficients obtained by predetermined learning with all pixels which are not classified, 
may be output. 

[0208] As described above, based on waveform characteristics of SD pixels constituting each tap, SD pixels con- 
stituting the tap are determined. Therefore, class taps for performing classification in which properties of pixels are 
reflected, and prediction taps in which prediction en-ors in predicted values are reduced, can be generated. As a result, 

10 the performance of the classification adaptive processing can be increased. 

[0209] The above-described processes shown in Rgs. 4. 6. 8. 1 0, 1 6, 1 7, 1 9. 21 , and 24 may be performed by hard- 
ware or by software. For using software to perfonn each process, a program for the software is installed in a computer 
built into an image processing apparatus and a learning apparatus as dedicated hardware, or a various types of pro- 
gram are installed in a multi-purpose computer that performs various types of processes. 

15 [0210] With reference to Figs. 25A, 25B, and 25C, a medium used when the programs for executing each of the 
above-described processes is installed in a computer beforehand so as to be executed by the computer are described 

below. « . 

[0211] The program can be provided to a user in the form that it is installed in a hard disk (indicated by "HDD" in 
Figs. 25A and 26) 102 or a semiconductor memory 1 03 as a recording medium built into a computer 1 01 , as shown in 
20 Fig. 25A. 

[0212] Also, by temporarily or permanently storing the program in a recording medium such as a floppy disk 1 1 1 , a 
compact-disk read-only memory (CD-ROM) 1 1 2, a magneto-optical (MO) disk 11 3. a digital versatile disk (DVD) 1 1 4, a 
magnetic disk 1 1 5, or a semteonductor memory 1 1 6, the program can be provided as package software, as shown in 
Fig.25B. 

25 [0213] In addition, as shown in Rg. 25C, by using radio communication to transfer the program from a download 
site 1 21 to a computer 1 01 via a digital broadcasting. satellite 122, or transferring, by wire, the program via a network 
131 such as the Internet, the program can be stored in the hard disk 102 built into the computer 101 . 
[0214] The "medium" in this specification means a broad concept including all the above-described types of 
medium. 

30 [021 5]. In this specification, the steps that describe the program provided by the medium does not need to be time- 
sequentially executed in accordance with the order indicated by each flowchart, but include processes (e.g.. parallel 
processing or object-oriented processing) executed in parallel or separately. 

[0216] With reference to Fig. 26, the structure of the computer 101 shown in Figs. 25A to 25C is described below. 
[0217] The computer 101 includes a central processing unit (CPU) 142, as shown in Fig. 26. An input/output (I/O) 

35 interface 1 45 is connected to the CPU 1 42 via a bus 1 41 . When a user inputs a command to the CPU 1 42 by operating 
an input unit 147 including a keyboard and a mouse, the CPU 142 executes, in accordance with the command, a pro- 
gram stored.in a read-only memory (ROM) corresponding to the semiconductor memory 103. The CPU 142 loads, into 
a random access memory 144, a program stored in the haftJ disk 102, a program installed in the hard disk 102 after 
being transferred from the satellite 1 22 or the network 133 and being received by a communication unit 1 48, a program 

40 installed in the hard disk 1 02 after being read from the floppy disk 1 1 1 , the CD-ROM 1 1 2, the MO disk 1 1 3, the DVD 
1 14, or the magnetic disk 115, which is loaded into a drive 149. The CPU 142 executes the loaded program, and out- 
puts the execution result to a display unit 1 46 including a liquid crystal display via, for example, the I/O interface 1 45, -as 
required. 

[0218] Since the classification adaptive processing uses teacher data and student data to perfonn learning for pre- 
45 diction coefficients corresponding to classes, and finds predicted values of the teacher data from input data by perfonn- 
ing linear prediction using the prediction coefficients and the input data, prediction coefficients for the desired predated 
values can be obtained depending on teacher data and student data for use in learning. By way of example, by using, 
as teacher data, an image including no noise, and using, as student data, an image generated by adding noise to the 
image as teacher data, predtetion coefficients for deleting the noise can be obtained. Accordingly, the present invention 
so can be applied not only to the above-described case in whfch resolution is increased by reducing blurring, but also to 
cases in which noise is eliminated and in which waveform is changed. 

[0219] In the foregoing embodiments, a moving picture is processed by the classification adaptive processing. 
However, in addition to the moving picture,, a still image, a sound, a signal read from a recording medium, etc.. can be 
processed. 

55 [0220] In the foregoing embodiments, class taps and prediction taps are generated in accordance with the same 
tap infonnation, and both types of taps are fomried by the same pixels. However, the class taps and the prediction taps 
can be generated so as to have different structures,, in other words, the class taps and the predtetion taps may be gen- 
erated in accordance with different types of tap inforrnation. 
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[02211 'n the foregoing embodiments, both class taps and prediction taps are generated in accordance with tap 
information, using SD pixels variably positioned to each pixel of interest. However, either type of taps may be generated 
using SD pixels fixedly positioned to each pixel of interest. 

[0222] In the second embodiment, SD pixels constituting taps are determined based on whether the SD pixels con- 
5 stituting the taps are extreme, the dynamic range, an error from an approximate line, etc. However, based on charac- 
teristics (waveform characteristics) of SD pixels in waveform, SD pixels constituting the taps may be detemnined, 
[02231 >n the second embodiment, SD pixels constituting taps are detemiined for directions from a center pixel as 
a start point. However, the SD pixels constituting taps may not be always determined for the directions. 
[0224] In the foregoing embodiments, an image processing apparatus and a learning apparatus for use in the ... 
10 image processing apparatus are separately provided. However, the Image processing apparatus and the learning appa- 
ratus may be integrally provided. In this case, learning can be perfomied in real time by the learning apparatus, and pre- 
diction coefficients for use in the image processing apparatus can be updated in real time. 

[0225] In the second embodiment, prediction coefficients for classes are prestored in the coefficient RAM 5. How- 
ever, the prediction coefficients may be supplied to the image processing apparatus, with, for example, input images 
IS (SD image). 

[0226] Class taps and prediction taps may be generated using either spatially positioned pixels and temporally 
positioned pixels. 

[0227] In the learning process, addition con-esponding to summation (L) in the expressions (7) is performed using 
prediction taps. Addition with prediction taps formed by SD pixels having different positions from each pixel of interest 

20 is performed using corresponding pixels in the prediction taps. 

[0228] In other words, when a basic tap is formed using 25 pixels composed of five horizontal pixels x five vertical 
pixels from a center pixel, a prediction tap is fomied using arbitrary pixels in the sixteen directions D^ to D^e shown in 
Rg. 14. Thus, a prediction tap and another prediction tap are fomied with pixels having different positions from the 
center pixel. In this case, a pixel as the first tap from the center pixel in the direction Di in a prediction tap, is added to 

25 a pixel as the first tap from the center pixel in the direction Dte «n another prediction tap. 

[0229] Also the classification is performed in the same aspect. In other words, when (the pixel level of) each pixel 
as a tap in each direction which constitutes a class tap is equal to each pixel as a tap in the con-esponding order from 
the center pixel in the corresponding direction, classification results based on the two class taps coincide with each 
other (are classified into the isame class). 

30 [0230] In so far as the embodiments of the invention described above are implemented, at least In part, using soft- 
ware-controlled data processing apparatus, it will be appreciated that a computer program providing such software con- 
trol, and a storage medium by which such a computer program is stored are envisaged as aspects of the present 
invention. 

35 Claims 

1. A data processing apparatus for predicting output data corresponding to input data by processing said input data, 
comprising: 

40 determining means capable of determining, in accordance with a plurality of data to be extracted from said 

input data, in-egular intervals among said plurality of data to be extracted from said input data; 
extracting means for extracting the plurality of data con-esponding to output data of interest to be predicted from 
said input data in accordance with the determination result by said determining means; and 
predicting means for finding predicted valueg of the output data of interest based on the plurality of data 

45 extracted by said extracting means. 

2. A data processing apparatus according to Claim 1 , wherein said determining means determines said plurality of 
data to be extracted from said input data so that a statistical value based on the plurality of data is approximately a 
predetermined value. 



50 



55 



3. A data processing apparatus according to Claim 1 , further comprising classifying means for classifying the output 
data of interest based on the extracted data and for outputting corresponding class codes, 

wherein said predicting means predicts the output data of interest by using prediction coefficients corresponding to 
the class codes. 

4. A data processing apparatus according to Claim 2, wherein said determining means determines said plurality of 
data to be extracted from said input data so that standard deviations based on the extracted data are approximately 

. a predetermined value. 
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5. A data processing apparatus according to Clainn 4, wherein said determining means determines the plurality of 
data to be extracted from said predetermined block based on the average of the standard deviations of said plurality 
of data to be extracted from said input data, which are located in a predetermined block of said input data. 

5 6. A data processing apparatus according to Claim 2, further comprising storage means for storing said predeter- 
mined value. 

7. A data processing apparatus according to Claim 3, wherein said predicting means uses said prediction coefficients 
and the extracted data to perform linear prediction of the output data of interest. 

10 

8. A data processing apparatus according to Claim 3. further comprising prediction-coefficient storage means for stor- 
ing said prediction coefficients corresponding to the class codes. 

9. A data processing apparatus according to Claim 1 , wherein said input data and the output data are image data. 

IS 

10. A data processing apparatus according to Claim 9, wherein said extracting means extracts, from said input data as 
said image data, pixels which are temporally and spatially adjacent to the output data of interest as said image 
data. 

20 11. A data processing apparatus according to Claim 9, wherein: 

said input data are image data having different degrees of blurring; and 

said determining means determines said plurality of data to be extracted from said input data for the respective 
image data having different degrees of blurring. 

,25 

12. A data processing apparatus according to Claim 9, wherein: . , 

said input data are image data having different resolutions; and 

said determining means determines said plurality of data to be extracted from said input data for the respective 
30 image data having different resolutions, 

13. A data processing apparatus according to Claim 9, wherein said determining means determines a width used for 
decimating a plurality of pixels as said plurality of data to be extracted from said input data as said image data so 
that a statistical value based on the pixels is approximately a predetermined value. 

35 

14. A data processing apparatus according to Claim 1, wherein said determining means determines said plurality of 
data to be extracted from said input data based on waveform characteristics of said plurality of data to be extracted 
from said input data. 

40 15. A data processing apparatus according to Claim 14, wherein said determining means detects an extreme value of 
said input data, and detennines, based on the detection result, said plurality of data to be extracted from said input 
data. 

16. A data processing apparatus according to Claim 14, wherein said detemiining means calculates a difference 
45 among said input data, and determines, based on the calculation result, said plurality of data extracted from siaid 

input data. 

17. A data processing apparatus according to Claim 14, wherein said determining means computes an en-or of said 
input data from a function approximating said input data, and determines, based on the computation result, said 

50 plurality of data to be extracted from said input data. 

18. A learning apparatus for predicting output data corresponding to input data by processing said input data, compris- 
- ing: 

55 determining means capable of detemiining, in accoixJance with a plurality of data to be extracted from said 

input data, in-egular intervals among said plurality of data to be extracted from said input data; and 
extracting means for extracting the plurality of data con-esppnding to output data of interest to be predicted from 
said input data in accordance with the determination result; and 
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computing means for finding prediction coefficients based on the extracted data. 

19. A learning apparatus according to Claim 18. wherein said. determining means determines the plurality of data 
extracted from said input data so that a statistical value of the plurality of data is approximately a predetermined 

5 value. 

20. A learning apparatus according to Claim 18, further comprising classifying means for outputting class codes corre- 
sponding to output data of interest by classifying said output data of interest based on the plurality of data extracted 
from said extracting means, said computing means finding the prediction coefficients corresponding to the class 

10 codes. 

21. A learning apparatus according to Claim 19, wherein said determining means determines the plurality of data 
extracted from said input data so that standard deviations based on the plurality of data extracted from said input 
data is approximately a predetermined value. 

15 

22. A learning apparatus according to Claim 21, wherein said determining means determines the plurality of data 
extracted from said input data based on the average of the standard deviations based on the plurality of data 
extracted from said input data. 

20 23. A learning apparatus according to Claim 19, further comprising storage means for storing said predetermined 
value. 

24. A learning apparatus according to Claim 1 9, further comprising predetermined-value computing means for comput- 
ing said predetemrilned value. 

25. A learning apparatus according to Claim 24, wherein said predetermined-value computing means uses the predic- 
tion coefficients for classes which correspond to the plurality of data to find, as said predetermined value, a statis- 
tical value of the plurality of data, which is obtained when results of processing said input data approach said output 
data. 

30 

26. A learning apparatus according to Claim 18, wherein said computing means finds prediction coefficients enabling 
prediction of the output data of interest by linear prediction using the plurality of data. 

27. A learning apparatus according to Claim 1 8. wherein said input data and the output data are image data. 
35 . 

28. A learning apparatus according to Claim 23, wherein said extracting means extracts, from said input data as said 
image data, pixels which are temporally and spatially adjacent to the output data of interest as said image data. 

29. A learning apparatus according to Claim 27, wherein: 

40 

said input data are image data having different degrees of blurring; and 

said determining means detemnines, for the respective image data having different degrees of blurring, the plu- 
rality of data extracted from said input data. ' ■' ^ 

45 3D. A learning apparatus according to Claim 27, wherein: 
said input data have different resolutions; and 

said determining means detemni nes, for the respective image data having different degrees of blumng. the plu- 
rality of data extracted from said input data. 

50 

31. A learning apparatus according to Claim 27, wherein said determining means determines a width for decimating a 
plurality of pixels extracted from said input data as said image data so that a statistical value based on the extracted 
pixels is approximately a predetermined value. 

55 32; A learning apparatus according to Claim 27. wherein said determining means detennines irregular intervals among 
a plurality of pixels extracted from said input data as said image data so that a statistical value based on the 
extracted pixels is approximately a predetennined value. 
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33. A learning apparatus according to Clainn 18, wherein said determining means determines the plurality of data 
extracted from said input data in accordance with waveform characteristics of the extracted data. 

34. A learning apparatus according to Claim 33, wherein said determining means detects an extreme value of said 
5 input data, and determines, based on the detection result, the plurality of data extracted from said input data. 

35. A learning apparatus according to Claim 33, wherein said determining means calculates a difference among said 
input data, and determines, based on the calculation result, the plurality of data extracted from said input data. 

10 36. A learning apparatus according to Claim 33, wherein said determining means computes an error of said input data 
from a function approximating said input data, and determines, based on the computation result, the plurality of 
data extracted from said input data. 

37. A data processing method for predicting output data corresponding to input data by processing said input data, said 
15 data processing method comprising the steps of: 

enabling determination of irregular intervals among a plurality of data to be extracted from said input data in 

accordance with said plurality of data to be extracted; 

extracting the plurality of data corresponding to output data of interest to be predicted from said input data in 
20 accordance with the detemnination result; and 

finding predicted values of the output data of interest based on the extracted plurality of data. 

38. A learning method for predicting output data corresponding to input data by processing said input data, said learn- 
ing method comprising the steps of: 

25 

enabling determination of irregular intervals among a plurality of data tp be extracted from said input data in 
accordance with said plurality of data to be extracted; 

extracting the plurality of data corresponding to output data of interest to be predicted from said input data in 
accordance with the detemnlnation result; and 
30 finding prediction coefficients based on the extracted plurality of data. 

39. A storage medium containing a computer-controllable program for predicting output data corresponding to input 
data by processing said input data, said computer-controllable program comprising the steps of: 

35 enabling determination of irregular intervals among a plurality of data to be extracted from said input data in 

accordance with said plurality of data to be extracted; 

extracting the plurality of data corresponding to output data of interest to be predicted from said input data in 
accordance with the determination result; and ^ 

finding predicted vajues of the output data of interest based on the extracted plurality of data. 

40 

40. A storage medium containing a computer-controllable' program for predicting output data corresponding to input, 
data by processing said input data, said computerTControilable program comprising the steps of: 

enabling determination of irregular intervals among a plurality of data to be extracted from said input data in 
45 accordance with said plurality of data to be extracted; 

extracting the plurality of data corresponding to output data of interest to be predicted from said input data in 

accordance with the determination result; and 

finding prediction coefficients based on the extracted plurality of data. 

50 
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FIG. 6 
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